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Look  and  Feel: 

Haptic  Interaction  for  Biomedicine 

1996  Annual  Report 


1  Introduction 


Figure  1:  The  Virtual  Surgery  Simulator  allows  the  user  to  see,  feel,  hear  and  manipulate  simulated  human 
organs. 

People  use  their  hands  to  touch,  manipulate,  and  learn  about  the  world  around  them.  Interaction 
through  touch  and  manipulation  is  known  as  haptic  interaction.  We  are  working  to  develop 
advanced  haptic  technology  that  allows  human  users  to  touch,  feel,  grasp,  and  manipulate  a  set 
of  special  objects:  objects  located  remotely,  objects  too  small,  too  large,  or  too  dangerous  for 
normal  human  interaction,  or  virtual  objects  that  exist  only  in  simulated  worlds  (Figure  1.) 

Advanced  haptic  interaction  will  have  a  wide  range  of  applications  in  biomedical,  military, 
and  civilian  areas.  It  will  allow  design  engineers  to  “pick  up”  new  designs,  “try  them  out,” 
and  see  how  they  behave,  before  they  ever  leave  the  drawing  board.  Engineers  will  sculpt  the 
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shapes  of  new  parts  in  the  computer,  as  though  made  of  clay.  Remote  experts  will  aid  the 
local  novice  in  performing  delicate  procedures.  Medical  students  will  train  by  manipulating 
virtual  organs,  without  risk  to  human  patients  or  animals.  Students  will  probe  any  part  of 
the  virtual  patient  with  impunity  and  view  a  wide  range  of  conditions  from  any  vantage  point. 
This  work  will  enable  applications  ranging  from  familiarization  and  training  in  domains  requiring 
hand/eye  skills  (surgery,  mechanics,  maintenance,  for  example)  to  the  enhancement  of  operations 
in  remote,  scaled,  unfamiliar,  and  hazardous  environments. 

Our  goal  is  to  develop  the  technological  foundation  of  advanced  haptic  systems  and  to  assem¬ 
ble  that  technology  into  a  useful  surgical  simulator.  The  surgical  simulator  we  envision  will  allow 
training  of  biomedical  procedures  in  a  virtual  environment  that  features  realistic  interaction  with 
simulated  anatomy  and  surgical  tools  through  haptic  force  feedback,  3D  graphics,  and  sound. 
In  addition  to  direct  training,  surgical  simulators  could  be  used  to  assess  surgical  performance 
and  play  a  role  in  certification.  This  report  describes  our  progress  in  building  such  a  system. 

In  the  first  year  of  this  effort  we  focused  on  developing  the  basic  technology  and  methods 
needed  to  build  interactive  haptic  applications.  In  this,  the  second  year  of  effort,  we  have 
assembled  these  components  into  several  prototype  systems.  We  have  built  integrated  systems  for 
surgical  anastomosis,  heart  palpation  and  aircraft  maintenance.  These  systems  were  built  upon  a 
common  software  foundation  that  includes  modules  for  virtual  haptic  interaction,  physics-based 
simulation,  and  computer  graphics.  While  these  systems  are  not  yet  complete  they  demonstrate 
significant  progress  towards  our  goal  of  building  virtual  reality  based  training  systems  that  are 
realistic,  flexible,  useful,  and  commercially  viable. 

2  Integrated  Haptics  Applications 

We  are  building  a  series  of  advanced  haptic  systems  that  consist  of  a  mechanical  force-feedback 
device,  a  real-time  computing  system,  a  dynamic  simulation  system,  and  computer  graphics 
and  sound(Figure  2).  Next  we  describe  these  systems:  a  simulator  for  surgical  anastomosis,  a 
demonstration  of  virtual  heart  palpation  that  uses  volumetric  3D  visual  displays,  and  a  simulator 
for  aircraft  maintenance.  While  the  functionality  of  each  of  these  systems  is  significantly  different, 
they  all  build  upon  a  common  foundation  and  thus  contribute  to  the  overall  goal  of  developing 
advanced  haptic  systems. 

2.1  The  Virtual  Surgery  Simulator 

The  primary  focus  of  activity  of  the  “Look  and  Feel”  project  remains  the  development  of  the 
Virtual  Surgery  Simulator  (Figure  1.)  This  simulator  is  an  integrated  system  that  allows  users 
to  reach  into  a  virtual  human  body  to  touch,  feel,  grasp  and  suture  two  deformable  tubes.  The 
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Figure  2:  Haptic,  Graphic,  and  Audio  Presentation  of  a  Virtual  Object 


user  holds  real  surgical  tools  connected  to  Phantom  force  feedback  devices.  By  holding  a  needle 
holder  in  one  hand  and  forceps  in  the  other,  the  user  can  manipulate  two  virtual  tube  organs  and 
suture  them  together  to  practice  an  end-to-end  anastomosis.  The  forces  of  interaction  between 
the  tools  and  the  simulated  tubes  are  reflected  to  the  user  with  the  Phantom  while  the  virtual 
workspace  is  visually  displayed  using  3D  computer  graphic  images.  Coordinated  visual  and 
tactile  displays  allow  the  user  to  work  in  an  intuitive  way  while  exploring  the  basic  techniques 
of  the  surgical  anastomosis. 

The  simulated  elements  in  the  anastomosis  demonstration  are  two  deformable  tubes,  forceps, 
needle  holder,  needle  and  suture  (Figure  3.)  The  look  and  feel  of  the  tubes  can  be  changed  to 
represent  different  anatomical  elements.  Tube  diameter,  thickness,  length,  compliance,  surface 
friction,  and  resistance  to  puncture  are  just  a  few  parameters  that  can  be  used  to  accomo¬ 
date  different  surgical  applications.  Texture  mapped  graphics  derived  from  photographs  of  real 
biological  tissue  help  make  the  simulated  elements  look  realistic. 

We  have  constructed  a  Virtual  Surgery  Station  that  allows  the  surgeon  and  surgical  subject 
to  be  in  the  correct  relative  position  (Figure  1.)  The  user  stands  with  hands  at  table  height 
holding  the  forceps  and  the  needle  holder.  Looking  down  at  his  hands,  the  surgeon  instead  sees  a 
mirror  reflecting  the  computer  graphic  image  of  the  subject  and  the  virtual  tools.  This  image  is 
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Figure  3:  The  Virtual  Surgery  Simulator  features  deformable  tubes  that  can  be  sewn  together  using  needle, 
suture,  needle  holder,  and  forceps. 

produced  in  approximately  the  correct  location  by  using  an  SGI  monitor  mounted  overhead.  A 
hand  rest  allows  the  surgeon  to  stabilize  his  hands  for  delicate  manipulation.  We  have  delivered 
a  version  of  this  Virtual  Surgery  Station  to  Dr.  Tom  Krummel  at  Pennsylvania  State’s  Hershey 
Medical  Center.  Dr.  Krummel  intends  to  integrate  this  trainer  into  his  virtual  training  lab 
at  the  Hershey  Medical  Center.  We  have  also  demonstrated  this  system  at  the  meeting  of  the 
Association  of  Academic  Health  Centers. 

2.2  Palpation  of  a  Beating  Heart 

Boston  Dynamics  Inc.  in  conjunction  with  Digital  Media  Associates  has  created  a  system  that 
allows  the  user  to  touch,  poke  and  pluck  a  simulated,  beating  heart  with  medical  instruments. 
The  user  can  touch  a  3D  volumetric  display  of  the  heart  with  medical  instruments  mounted  to 
Phantom  force-feedback  devices.  The  user  actually  feels  the  heart  whenever  the  physical  tool 
“touches”  the  floating  3D  image  because  the  visual  and  haptic  images  of  the  heart  are  coincident 
in  space.  An  audible  heart  beat  is  synchronized  with  the  visual  and  haptic  heart  displays. 

The  volumetric  3D  display  is  created  by  projecting  computer  graphic  images  of  a  beating 


Figure  4:  The  Virtual  Heart:  the  user  can  see,  feel,  and  hear  the  heart  beat  as  well  as  poke  and  pluck  its 
deformable  surface. 

heart  onto  a  screen  contoured  to  the  shape  of  a  heart.  The  image  of  this  screen  is  projected  into 
the  haptic  workspace  using  mirrors.  The  projection  system,  the  3D-RT,  was  built  by  Digital 
Media  Associates.  It  consists  of  a  high  quality  projector,  the  heart  contoured  screen,  and  the 
mirroring  system.  BDI  created  the  haptic  algorithms,  dynamic  simulation,  computer  graphics, 
and  sound  of  the  beating,  deformable  heart.  This  unique  combined  display  system  makes  it 
easier  to  touch  virtual  objects  by  putting  haptic  and  visual  images  together. 

2.3  Virtual  Aircraft  Maintenance  Training 

The  technology  that  provides  the  foundation  for  advanced  haptics  systems  is  largely  indepen¬ 
dent  of  the  application  area.  In  a  separately  funded  project  we  developed  the  Virtual  Aircraft 
Maintenance  Trainer.  This  project  has  leveraged  the  funding  for  surgical  simulator  development 
because  the  basic  issues  regarding  dynamic  simulation,  haptic  force-feedback,  and  computer 
graphics  are  common  to  all  haptics  systems. 

We  developed  the  Virtual  Aircraft  Maintenance  Trainer  to  explore  the  feasibility  of  replac¬ 
ing  expensive  mock-ups  of  aircraft  used  for  maintenance  training  with  less  expensive  computer 


Figure  5:  The  Virtual  Maintenance  Trainer  allows  the  user  to  push  buttons,  throw  toggle  switches,  turn  dials, 
observe  lights,  and  listen  to  audible  tones,  and  manipulate  linkages  during  the  evaluation  and  adjustment  of 
aircraft  systems. 

hardware  and  software.  The  virtual  maintenance  trainer  we  built  allows  a  user  to  perform  the 
essential  elements  of  two  real-world  maintenance  training  tasks  performed  by  the  US  Marines 
on  the  AV8B  aircraft,  a  vertical  take-off  and  landing  attack  fighter.  One  task  is  diagnosis  of  a 
failed  Radar  Warning  Receiver  System  Built-In-Test  (RWR  BIT).  The  second  maintenance  task 
is  adjustment  of  the  Vernier/Non- Linear  Nosewheel  Steering  Linkage. 

Using  the  Virtual  Maintenance  Trainer  the  trainee  can  push  buttons,  throw  toggle  switches, 
and  turn  dials  in  the  cockpit,  as  well  as  observe  indicator  lights,  listen  for  audible  tones  and 
measure  voltages  as  part  of  running  and  diagnosing  an  avionics  self-test.  Simulations  of  the 
cockpit  panels,  buttons,  dials,  switches,  and  electrical  pins  allow  the  technician  to  see,  feel,  and 
hear  the  avionics  system  operations.  The  linkage  adjustment  procedure  involves  manipulating  a 
simulated  four-bar  linkage  through  its  range  of  motion,  measuring  the  forces  required  to  move 
the  linkage  between  two  hard  stops,  and  adjusting  those  forces  by  tightening  or  loosening  a 
friction  nut  on  the  linkage.  The  dynamic  simulation  of  the  linkage  obeys  the  kinematic  and 
dynamic  constraints  of  the  real  physical  device  and  allows  interactive  attachment  or  detachment 
from  the  rudder  pedal  actuator,  and  joint  friction  adjustment.  The  user  can  feel  the  shapes  of 
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the  links  in  the  mechanism,  as  well  as  the  forces  needed  to  move  it. 

3  Progress  on  Components  of  an  Advanced  Haptic  System 

An  integrated  system  for  advanced  haptic  interaction  needs  to  have  three  parts:  a  part  that 
mechanically  interfaces  to  the  user’s  hand  (and/or  arm),  a  part  that  mediates  the  interaction 
between  the  hand  and  virtual  or  remotely  located  objects,  and  a  part  that  simulates  how  the 
objects  will  behave  when  touched.  These  components  are  force  feedback  devices,  haptic  algo¬ 
rithms,  and  physics-based  simulation,  respectively.  In  Table  1  we  summarize  our  progress  on  a 
list  of  development  tasks  for  advanced  haptic  systems  including  the  virtual  surgical  simulator. 
Next  we  detail  our  progress  in  these  areas. 

3.1  Advanced  Haptic  Interface  Devices 

The  Phantom  haptic  device  is  a  light  weight,  high  bandwidth  mechanical  interface  that  allows 
the  user  to  touch  and  manipulate  a  virtual  or  remotely  located  object.  Haptic  fidelity  and 
high  resolution  are  important  for  conveying  a  detailed  sense  of  touch  and  are  thus  emphasized 
over  kinematic  complexity  and  high  force  capacity.  High  fidelity  and  resolution  are  obtained 
by  maximizing  bandwidth  through  the  design  of  stiff,  lightweight,  and  low  friction  structures 
and  transmissions.  The  Phantom  is  interfaced  to  both  a  PC  and  an  SGI  graphics  workstation. 
Haptic  algorithms  and  object  simulation  code  are  run  on  the  PC  while  the  graphics  workstation 
is  used  for  3D  visual  display. 

The  Phantom  is  both  an  input  device  and  an  output  device.  It  can  apply  forces  back  to 
the  user  along  the  three  translational  axes.  The  position  and  orientation  of  the  tool  or  probe 
mounted  to  its  tip  are  measured  using  encoders.  Instrumented  surgical  tools  or  mechanical  tools 
mounted  to  the  Phantom  provide  a  familiar  interface  to  the  user.  A  powered,  two  finger  device 
has  been  designed  by  the  MIT  group  and  tested  at  BDI.  This  device  can  apply  a  gripping  torque 
to  the  user  to  simulate  the  sensation  of  grasping  an  object  with  two  fingers. 

Further  development  of  the  haptic  device  is  underway  and  will  include  piezo- vibration  stim¬ 
ulators  for  ultra-high  bandwidth  sensation,  multi-fingertip  interaction,  and  large  workspace. 

3.2  Haptic  Algorithms 

Haptic  algorithms  mediate  the  exchange  of  forces  between  simulated  objects  and  the  user.  The 
two  most  important  haptic  algorithms  are  for  contact  detection  and  contact  force  modeling. 
Contact  detection  is  used  for  determining  when  two  simulated  objects  are  touching.  Contact 
force  models  are  used  to  compute  the  forces  of  interaction  between  simulated  objects.  Contact 
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Table  1:  Progress  List  for  Advanced  Haptic  System  Development 


•  Advanced  Haptic  Interface  Devices:  The  haptic  device  is  the  mechanical  interface  that  allows  the  user  to 
touch  and  manipulate  a  virtual  or  remotely  located  object. 

—  Phantom  haptic  device  interfaced  to  SGI  and  PC 

—  Instrumented  medical  tool  attachments 

*  Debakey  forceps 

*  needle  holder 

—  Virtual  medical  tools 

*  debakey  forceps 

*  needle,  needle  holder,  and  suture 

*  scalpel  with  scalpel  holder 

*  hemostatic  forceps 

*  intestinal  forceps 

*  dissecting  forceps 

*  scissors 

*  retractor 

—  Instrumented  gimbal  measures  tool  position  and  orientation 

—  Powered,  two  finger  attachment  designed  and  tested  (MIT  Task) 

•  Haptic  Algorithms:  These  algorithms  mediate  the  forces  the  user  feels  with  the  Haptic  Interface  Device. 

—  Two  handed  interaction  demonstrated 

—  Two  finger  manipulation  demonstrated 

—  Rigid  body  contact  force  models 

—  Flexible  body  contact  force  models 

—  Coulomb  friction 

—  Haptic  textures  for  bumps,  sand  paper,  ridges 

—  Haptic  textures  mapped  onto  shapes 

—  Contact  detection  between  3D  polygonal  shapes 

—  Contact  detection  between  point  or  line  and  flexible  tubes 

—  Contact  detection  between  point  and  grid  maps 

•  Physics-Based  Simulation:  Real-time  physics-based  simulation  is  used  to  calculate  the  behavior  of  objects, 
interactions  among  objects,  and  interactions  between  objects  and  the  user 

—  Automatic  simulation  generation  for  rigid  body  systems 

—  Loadable  object  shapes  from  CAD  files 

—  Programmable  bulk  properties  of  simulated  objects 

—  Compliant  flexible  skin 

—  Compliant  flexible  tubes 

—  Calcified  artery 

—  Compliant  beating  heart 

—  Coordinated  3D  volumetric  display  and  haptic  display 

—  Suture  two  tubes  together  with  surgical  needle  and  suture 

—  Texture  mapped  graphics 
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forces  can  arise  from  interpenetration  of  objects,  sliding  between  objects,  or  from  haptic  textures 
applied  to  surface  geometry. 

3.2.1  Contact  Detection 

The  contact  detection  algorithm  determines  when  simulated  objects  collide  or  when  the  user 
is  touching  a  simulated  object.  The  algorithms  use  the  geometry  of  the  simulated  objects  to 
compute  the  distance  separating  them.  We  use  several  contact  detection  algorithms  that  are 
optimized  for  real-time  performance  and  that  are  tuned  to  the  object  model  type.  For  example, 
we  use  a  linear-time  Lin  and  Canny  type  algorithm  for  contact  between  convex  rigid  polyhedra. 
This  general  contact  algorithm  is  insensitive  to  the  number  of  polygons  on  an  object.  We  use  a 
grid-depth  type  algorithm  for  point  contact  on  polyhedra  with  mild  concavities  and  a  specialized 
algorithm  for  computing  contact  between  points  or  lines  and  a  flexible  tube. 

3.2.2  Contact  Force  Models 

The  interaction  force  between  two  touching  simulated  objects  is  computed  with  a  contact  force 
model.  We  use  penalty  method  techniques  in  our  contact  force  models.  Characteristics  such 
as  surface  compliance,  friction,  textures,  puncturability,  and  surface  feature  resolution  are  con¬ 
trolled  by  the  contact  force  model  and  can  be  modified  to  suit  the  application. 

3.2.3  Haptic  Texture  Mapping 

The  goal  of  haptic  texture  mapping  is  to  permit  haptically  perceivable  textures  to  be  overlaid 
on  basic  object  geometry.  A  haptic  texture  is  a  force  model  whose  parameters  depend  upon  the 
spatial  location  of  the  haptic  device  relative  to  the  underlying  geometry.  Our  basic  technique 
is  to  superpose  forces  due  to  haptic  textures  and  forces  due  to  contact  with  the  underlying 
geometry.  The  spatial  information  for  a  haptic  texture  is  derived  from  loadable  images  and  can 
be  used  to  change  different  parameters  in  the  contact  force  models.  Using  haptic  textures  we 
have  created  the  feeling  of  scratchy  sand  paper,  smooth  bumps,  and  discrete  ridges  on  a  flat 
surface.  We  are  currently  working  on  mapping  haptic  textures  to  rigid  objects  and  deformable 
organ  models. 

3.3  Physics-Based  Simulation 

Our  system  uses  real-time  physics-based  simulation  to  calculate  the  behavior  of  objects,  inter¬ 
actions  among  objects,  and  interactions  between  objects  and  the  user.  Objects  can  be  isolated 
bodies,  mechanical  linkages,  or  more  complicated  things  like  tool  mechanisms,  deformable  tis¬ 
sues,  or  organs.  A  basic  idea  of  the  work  is  that  physics-based  simulation  allows  virtual  objects 
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to  move  as  real  objects  do;  obeying  the  laws  of  physics,  they  feel  like  real  objects  when  touched, 
and  move  realistically  in  response  to  touching. 

The  foundation  of  our  object  simulator  is  software  that  we  developed  in  previous  work  that 
automatically  creates  highly-optimized  dynamic  simulations  of  things  that  move  from  simple 
descriptions  of  the  objects.  The  physical  simulations  include  material  properties  than  can  be 
changed  by  the  user  to  tune  the  feel  of  the  object.  For  example,  we  can  change  the  mass  of  a 
rigid  body,  the  compliance  of  its  surface,  or  the  coefficient  of  friction  between  a  surface  and  the 
haptic  device. 

3.3.1  Rigid  Object  Loader 

A  general  object  loader  is  working  that  allows  multiple  3D  objects  to  be  loaded,  dynamically 
simulated,  and  touched  or  manipulated.  The  software  includes  automatic  contact  detection, 
equation  generation,  and  3D  graphics.  The  shape  of  these  objects  can  be  defined  with  a  CAD 
file  that  describes  their  surface  geometry.  The  geometry  is  used  for  both  visualization  and  contact 
detection.  Bulk  properties  of  the  bodies  like  mass,  coefficient  of  friction,  or  surface  impedance 
can  be  specified  by  the  user.  The  haptic  tool  used  for  manipulation  in  the  virtual  environment 
is  a  special  kind  of  simulated  object.  It’s  position  and  orientation  are  driven  by  the  Phantom. 
The  forces  of  interaction  between  it  and  other  simulated  objects  are  computed  from  the  haptic 
algorithms. 

3.3.2  Flexible  Tissue 

To  support  the  simulation  of  surgical  procedures,  we  have  expanded  our  simulation  library 
to  include  prototype  simulations  of  flexible  tissue  and  flexible  tubes.  At  BDI  we  are  using 
flexible  tissue  simulation  techniques  that  strive  for  computational  simplicity  and  speed  while 
providing  convincing  behavior.  One  prototype  flexible  tissue  simulation  included  a  planar  surface 
that  deformed  when  pushed  or  pulled.  Rigid  objects  with  varying  surface  compliance  could  be 
positioned  beneath  the  flexible  tissue  to  simulate  a  palpation  task.  We  also  have  simulations  of 
whole  and  severed  hollow  tubes.  The  tubes  move  when  touched  and  the  surfaces  deform  when 
the  tubes  are  poked  or  plucked.  By  varying  the  radii  of  the  tubes  with  time  we  can  simulate  the 
look  and  feel  of  a  pulsing  vessel.  The  flexibility  or  surface  deformation  of  a  flexible  tube  can  be 
tuned  to  best  approximate  the  feel  of  biological  material. 

4  Commercialization  and  Related  Work 

BDI  is  developing  and  selling  products  that  are  closely  related  to  the  ARPA  funded  work. 
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•  BDI  is  marketing  Tangible  Reality(TM),  a  turn-key  virtual  environment  system  that  in¬ 
cludes  visual  images,  sound,  and  touch.  The  system  consists  of  a  force-feedback  device,  a 
real-time  simulation  computer,  a  physics-based  simulation  environment,  simulated  objects, 
real-time  3D  computer  graphics,  real-time  3D  sound  effects,  and  networking.  Early  versions 
of  the  product  will  support  standard  sets  of  physics-based  simulated  objects.  Later  versions 
will  support  user-defined  objects  and  deformable  objects.  This  product  provides  an  initial 
channel  for  commercialization  of  results  from  the  proposed  research.  For  example  “palpat¬ 
ing  in”  and  “haptic  texture  mapping”  are  obvious  candidates  for  enhancing  the  product  as 
we  understand  and  develop  them. 

•  We  delivered  a  special  version  of  Tangible  Reality  designed  for  training  surgical  procedures. 
This  version  of  the  system  supports  two-handed  anastomosis.  Thomas  M.  Krummel,  M.D., 
Chair  of  the  Department  of  Surgery  at  Penn  State’s  Hershey  Medical  Center  is  building 
“The  Virtual  Hospital.”  The  VR  Anastomosis  Simulator  is  a  key  feature  in  his  concept  for 
reducing  the  cost  and  improving  the  quality  of  medical  teaching. 

•  We  delivered  to  Digital  Media  Associates  a  system  that  allows  the  user  to  touch,  poke 
and  pluck  a  simulated,  beating  heart  with  medical  instruments.  The  user  can  touch  a 
3D  volumetric  display  of  the  heart  with  medical  instruments  mounted  to  Phantom  force- 
feedback  devices.  The  user  actually  feels  the  heart  whenever  the  physical  tool  “touches” 
the  floating  3D  image  because  the  visual  and  haptic  images  of  the  heart  are  coincident  in 
space.  An  audible  heart  beat  is  synchronized  with  the  visual  and  haptic  heart  displays. 
We  have  plans  to  continue  this  joint  project  to  combine  force  feedback  haptics,  dynamic 
simulation,  and  3D  volumetric  displays. 

•  We  have  developed  a  joint  project  with  Musculographics  Inc.  to  develop  a  trainer  for 
wound  debriedment  for  the  U.S.  Army  at  Fort  Bragg.  Initial  phases  of  this  project  involve 
development  of  a  leg  and  leg  wound  simulator  with  haptic  tool  interaction  for  training.  BDI 
will  provide  the  haptic  components  of  this  project.  A  version  of  this  simulator  is  included 
in  the  FY97  P.O.M. 

•  Nissho  Electronics  Corp.  is  a  Japanese  distributor  of  the  BDI  Tangible  Reality  System. 

•  In  an  effort  to  demonstrate  the  applicability  of  VR  technology  to  aircraft  maintenance  train¬ 
ing  we  developed  the  VR  Maintenance  Trainer  for  the  Joint  Advanced  Strike  Technology 
Program.  This  project  significantly  leveraged  our  ARPA-funded  project  and  offers  other 
military  applications. 
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5  Conclusions 

Two  years  into  the  “Look  and  Feel”  project  we  have  assembled  much  of  the  basic  technology 
required  to  build  integrated  haptics  systems  that  allow  the  user  to  see,  hear,  and  touch  simulated 
objects.  These  worlds  can  include  rigid  body,  multi-link,  dynamic  objects,  deformable  organs, 
and  articulated  tools  that  can  be  manipulated  by  the  user.  Haptic  algorithms  detect  collisions 
between  simulated  objects  and  compute  contact  forces  between  them.  Force-feedback  devices 
convey  the  forces  of  interaction  to  the  user.  Physics-based  simulation  computes  the  response  of 
simulated  objects  to  external  forces  and  3D  computer  graphics  display  the  visual  scene  to  the 
user.  Using  these  component  technologies  we  have  constructed  prototypical  advanced  haptic 
systems  that  allow  the  user  to  perform  procedures  such  as  surgical  anastomosis,  heart  palpation, 
and  aircraft  maintenance.  While  these  prototype  demonstrations  do  not  replicate  all  the  detail 
found  in  real  surgeries  or  training  environments,  we  believe  they  take  a  significant  step  towards 
the  goal  of  building  useful  virtual  training  environments. 

In  the  near  term  we  will  continue  to  work  towards  the  goal  of  producing  a  surgical  simulator  for 
training  or  evaluating  medical  personnel.  To  this  end  we  will  focus  on  making  the  interaction  with 
biomedical  objects  more  realistic  and  natural.  We  will  add  performance  measurement  and  display 
features  to  the  simulator.  These  recording,  display,  and  playback  features  will  help  the  surgical 
trainee  review  and  improve  surgical  technique  by  providing  quantitative  measures  of  needle 
placement  accuracy,  force  level,  and  tissue  damage.  Once  these  features  are  working,  the  user 
will  be  able  to  practice  important  elements  of  surgical  anastomosis,  including  proper  handling  of 
the  medical  tools,  techniques  for  grasping  tubes,  and  suturing  techniques  including  proper  needle 
orientation,  needle  placement,  puncture  forces,  suture  spacing,  and  suture  tightening.  We  will 
incorporate  new  features  into  the  simulator  such  as  haptic  textures,  and  a  set  of  re- configurable 
organs  that  allow  the  user  to  tune  the  look  and  feel  of  the  organ  to  suit  the  application. 

Additional  performance  from  virtual  training  applications  will  likely  require  more  complex 
object  simulations  and  contact  detection  software.  We  will  continue  to  optimize  and  refine  our 
contact  detection  algorithms,  dynamic  simulation  techniques,  and  computer  graphics  software  so 
that  we  can  do  more  with  the  computers  we  currently  use.  In  addition  to  making  it  possible  to 
create  more  realistic  virtual  environments,  more  efficient  software  will  allow  us  to  create  useful 
low-end  haptic  systems  that  run  on  inexpensive  computers. 
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